Add 'balancedweighted' VM allocation algorithm - #14109

Description

This PR adds an opt-in VM allocation algorithm, balancedweighted, for vm.allocation.algorithm. Existing algorithms and the default are untouched.

The problem. The existing algorithms rank hosts on allocated capacity alone. Under a large overprovisioning factor that reads badly: allocation is measured against a total that has already been multiplied by the factor, so a host under real strain can still report a low percentage allocated and keep attracting new VMs. Anything allocation cannot see - guests using far more than they asked for, VMs the management server has lost track of - is invisible.

Concurrent deployments make it worse. Capacity is only charged once a VM starts, so every decision taken in the same moment reads the same figures, and strict ordering makes them all agree on one host.

What it ranks on. A blend, lower is better:

Recent starts exist because a VM that started moments ago is invisible to both allocation lag and a moving average, while often working hardest.

A dominant-resource term is added on top of the weighted mean, taking the larger of allocated and measured per resource, so that a host nearly out of any one resource does not rank well on a good average.

Then two things happen that ranking alone does not do:

• hosts measurably too busy are held back, unless that would leave nowhere to deploy
• selection is random among the best few rather than strictly ordered, so simultaneous deployments do not all pick the same host

The algorithm

Every candidate host gets a score in [0, 1]; lower is better.

Terms. Each is a fraction of that host's capacity for the resource, so the weights are directly comparable:

cpu_alloc  = (used + reserved CPU)    / (raw total CPU    x cluster cpu overcommit ratio)
mem_alloc  = (used + reserved memory) / (raw total memory x cluster memory overcommit ratio)
cpu_util   = EWMA of measured CPU utilisation      (fraction of real cores)
mem_util   = EWMA of measured memory in use        (fraction of real memory)
vm_count   = VMs on the host            / host.weighted.expected.vms.per.host
recent     = VMs started within window  / host.weighted.expected.vms.per.host

All six are clamped to [0, 1]. The allocated terms divide by the overprovisioned total because that is what a host can hand out; dividing by the raw total makes every host on an overcommitted cluster read as full.

Score.

             w_ca*cpu_alloc + w_cu*cpu_util + w_ma*mem_alloc + w_mu*mem_util + w_vc*vm_count + w_rs*recent
  mean(h) = ---------------------------------------------------------------------------------------------
                              w_ca + w_cu + w_ma + w_mu + w_vc + w_rs

  dominant(h) = max( max(cpu_alloc, cpu_util), max(mem_alloc, mem_util) )

                mean(h) + w_dom * dominant(h)
  score(h)  =  ------------------------------
                        1 + w_dom

dominant is the host's most stressed resource, and it takes the larger of allocated and measured per resource — memory reclaimed from idle guests is taken back the moment those guests get busy, so the optimistic reading should not win. Adding it on top of the mean stops a host that is fine on average but nearly out of one resource from ranking well. Both parts are convex combinations of values in [0, 1], so the score stays in [0, 1].

When a host has no usable utilisation samples, w_cu and w_mu drop out of both numerator and denominator, and the host is ranked behind every host that can be measured rather than being assumed idle.

Moving average. Weighted by elapsed time, so a missed poll decays by the right amount instead of over-weighting the previous value:

  value  <-  value + a * (sample - value),    a = 1 - exp( -dt * ln2 / half_life )

Selection. Ranking alone is not enough: capacity is only charged once a VM starts, so concurrent deployments all read the same figures and strict ordering makes them agree on one host.

1. split measured hosts into healthy / over-threshold
2. sort each group by score, ascending
3. shuffle the first N healthy hosts        (N = host.weighted.selection.spread)
4. append hosts that cannot be measured
5. append hosts held back by a threshold    (used only if nothing above them fits)

Settings

Weights are relative to each other; only their ratios matter. 0 disables a term.

The first four are defined in api rather than alongside the allocator, because they describe how
loaded a host is rather than anything specific to placement, and a DRS algorithm that ranks hosts
the same way should read the same settings instead of carrying a second copy that could disagree.

All are dynamic except host.load.sample.interval.

Measured effect. Simulation over a churning, heavily overprovisioned fleet where part of the real load is invisible to allocation, measuring how unevenly real load ends up distributed (max/mean across hosts, lower is better):

The middle row is a control: it isolates what the scoring contributes from what the randomisation contributes.

Utilisation figures come from a moving average of what StatsCollector already polls but nothing used for placement. It is per management server and not persisted; every management server polls every host, so they converge. Until a server has samples, ranking falls back to allocation figures, and hosts that cannot be measured rank behind hosts that can rather than being assumed idle.

Note the utilisation terms are only meaningful as intended on KVM. getCpuUtilization reports a reservation figure on VMware and is scaled by core count on XenServer; this is documented on the collector.

All weights and thresholds are settings, most cluster scoped.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

How Has This Been Tested?

Unit tests, 39 new cases:

• WeightedHostScorerTest - the scoring function: each term's direction, the dominant-resource term, unmeasured hosts, the utilisation gates, the selection spread.
• WeightedHostScorerRankTest - rank() end to end as the allocator calls it, with the capacity and VM-count queries mocked. Covers the overprovisioned denominator, a busy host ranking behind a quiet one at equal allocation, an over-threshold host never leading while a healthy one exists, unmeasured hosts ranking last, fallback to allocation when nothing is measured, and a host missing a capacity row.
• HostLoadTrackerTest - the moving average: first sample, a single spike not dominating, convergence, half-life, missed samples decaying by elapsed time rather than sample count, unchanged readings not folded twice, and a host that stops reporting becoming unusable.
• WeightedPlacementDistributionTest - the simulation above. Deterministic; the workload is fixed before any arm runs so all arms see identical VMs.

Full mvn test on api, engine/schema and server, checkstyle and license checks enabled: 0 failures.

How did you try to break this feature and the system with this change?

This went through two rounds of adversarial review. The defects found and fixed are worth listing, since they are the interesting part:

• The allocated fraction was measured against the wrong total. op_host_capacity stores totals raw and applies overprovisioning when they are read. Dividing by the stored total made the fraction reach 1 at the host's physical size, so on a cluster overcommitted ten times every host clamped to 1 and both the allocation term and the dominant-resource term went dead - on exactly the clusters this is for.
• The utilisation threshold could be bypassed. Held-back hosts were re-appended before the random spread was applied, so with one healthy host and a spread of three, two thirds of deployments picked an over-threshold host.
• A host with no samples was treated as idle, exempt from the thresholds and scoring well on the dominant term - so a host with broken statistics became a deployment magnet.
• A host that stopped reporting kept vouching for itself. StatsCollector hands back the previous entry when a poll fails, and that unchanged reading was folded again every interval.
• The sampler ran on a Timer, which dies permanently and silently on one escaping error, leaving placement quietly back on allocation alone. It also ran regardless of whether the algorithm was selected.
• A negative weight would have ranked the most loaded host first.
• The simulation drew different random streams per arm, so the arms were not seeing the same workload; and it bypassed rank(), which is why it caught none of the above. Both fixed, and the spread-only control arm added.

Other things checked: hosts missing a capacity row, all-zero weights, podId/clusterId being null, concurrent access to the shared average, and that the two per-ranking queries became one.